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DETAILED ACTION 
Continued Examination Under 37 CFR 1.114 

1. A request for continued examination under 37 CFR 1.114, including the fee set 
forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this 
application is eligible for continued examination under 37 CFR 1.114, and the fee set 
forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action 
has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 

1 1/15/2006 has been entered. 

Response to Amendment 

2. This office action is in response to communications filed 1 1/15/2006. Claims 1 , 
14, 17, 18, 21, 35, 46, 47, 50-52, 56 are amended. Claims 2-9, 19, 20, 36-43, 48, 49 
are original. Claim 55 is previously presented. Claims 10-13, 15, 16, 22-34, 44, 45, 53, 
54, 57-60 are cancelled. Claims 61-62 are new. 

Response to Arguments 



3. Applicant's arguments with respect to claims have been considered but are moot 
in view of the new ground(s) of rejection. 
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Applicant's arguments filed 1 1/15/2006 have been fully considered but they are 
not persuasive. 

The applicant argues that Srinivasan does not disclose a coordinate tracking 
data including a mask data packet set for each video frame of the plurality of 
consecutive video frame... each mask data packet set including location and graphics 
data of a graphics image to be overlaid on the video object appearing in the 
corresponding vjdeo frame (Pages 12-13). The Examiner respectfully disagrees. 
Srinivasan disclose mask data or an authoring station profile mask data for each video 
frame of a plurality of consecutive video frames (Column 7, lines 22-25, Column 8, lines 
13-14, 38-49); and that each mask data includes location or coordinate and graphics 
data of a graphics image to be overlaid on the video object (Column 5, line 54-65, 
Column 6, lines 8-19, Column 7, lines 33-47). Srinivasan discloses tracking an object 
based on time and location for the movement from a first location to a second location 
(Figures 1-6, Figure 14, Figure 18). Srinivasan discloses that end user equipment has 
the hardware and software to synchronize data on a frame-by-frame basis based on 
annotation timing data (Abstract, Column 4, lines 35-37, 55-62, Column 5, line 54-65, 
Column 6, lines 8-19, Column 7, lines 21-47, Column 12, lines 21-46, Column 36, lines 
41-49, Column 21, lines 34-52). See rejections. 

4. The applicant argues that Wistendahl that the IDM program is not the claimed 
mask even though it contains graphics data (Page 11). 
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In response to the arguments, Srinivasan discloses all limitations of claim 1 (see 
above response to arguments). Wistendahl discloses a mask (Column 10, lines 8-35, 
Figure 5a) and that the outline data of one frame for succeeding frames for the object 
(Column 10, lines 57-61). See rejections below. The examiner would like to point out 
that Wistendahl is not used to teach the limitations of Claim 1 . 

5. The applicant argues that Shoff does not teach a video frame for plurality of 
consecutive video frames and overlaying two or more graphic images is synchronized 
on a frame-by-frame basis. Srinivasan discloses all the limitations of claim 1 . 

The examiner would like to point out that Shoff is not used to teach a video frame 
for plurality of consecutive video frames and overlaying two or more graphic images is 
synchronized on a frame-by-frame basis. Shoff is used to teach different limitations. 

Information Disclosure Statement 

6. The information disclosure statement filed 1-1 1-02, 5-13-04 fails to comply with 
37 CFR 1 .98(a)(2), which requires a legible copy of each cited foreign patent document; 
each non-patent literature publication or that portion which caused it to be listed; and all 
other information or that portion which caused it to be listed. It has been placed in the 
application file, but the information referred to therein has not been considered. 

The applicant has submitted many of the foreign patents and non-patent 
literature documents. However not all documents in the information disclosure 
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statement filed 1-11-02, 5-13-04 have been submitted including IBM Technical 
Disclosure Bulletin's "User Controlled Display of Hypermedia Links Via Mouse Location" 
and "Visual Behavior of Multimedia Hypervideo Links for Motion Video". 

7. The information disclosure statement (IDS) submitted 31 July 2006. The 
submission has met the minimum requirements of 37 CFR1.97 and 37 CFR 1.98. 
Accordingly, the examiner is considering the IDS. 

It is noted, however, that the large number of references, represents a significant 
burden for the examination of the instant application. Therefore, the references have 
only been considered to the extent possible given limited examination resources. 
Should the applicant be aware that one or more of these documents is particularly 
relevant to patentability, it is requested that applicant's provide a concise explanation of 
why the English language information is being submitted and how it is understood to be 
relevant. 

Claim Objections 

8. Claims 1-9,14, 17-21, 35-43, 46-52, 55, 56, 61, 62 are objected to because of the 
following informalities: The claims recite "tracing movement of two or more video 
objects.." The applicant's specification more clearly describes the function as "tracking" 
(Page 8, paragraph 0031). The Examiner suggests, "tracing" to be -tracking-. 
Appropriate correction is required. 
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Claim Rejections - 35 USC § 102 

9. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that 
form the basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - - 

(e) the invention was described in (1) an application for patent, published under section 122(b), by 
another filed in the United States before the invention by the applicant for patent or (2) a patent 
granted on an application for patent by another filed in the United States before the invention by the 
applicant for patent, except that an international application filed under the treaty defined in section 
351(a) shall have the effects for purposes of this subsection of an application filed in the United States 
only if the international application designated the United States and was published under Article 21(2) 
of such treaty in the English language. 

10. Claims 1-7, 17-20, 35-41 are rejected under 35 U.S.C. 102(e) as being 
anticipated by Srinivasan et al (US 6,357,042 and hereafter referred to as "Srinivasan"). 

Regarding Claims 1 and 35, Srinivasan discloses a hyperlinked broadcast 
system and a method of generating a hyperlinked video signal (Figure 1, Figure 14, 
Figure 18) comprising: a video headend or video source provides video information for a 
video program including a plurality of consecutive video frames (Figure 1, 12, Column 7, 
lines 22-25, Column 8, lines 13-14, 38-49), 

an annotation system tracing or tracking movement of two or more video objects 
appearing in each of plurality of consecutive video frames from a first location to a 
second location (Column 8, lines 5-37) and generating annotation data (Column 6, lines 
8-19) and annotation data timing information (Column 7, lines 21-30, Column 8, lines 5- 
50), the annotation data including only one mask for each video frame of the plurality of 
consecutive frames (Column 5, lines 54-65, Figure 3, Figure 4), each mask being 
stamped with the corresponding annotation data timing information based on a frame 
time of the corresponding video frame (Abstract, Column 7, lines 21-30, Column 8, lines 
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5-50), each mask including location (Figure 4, Column 8, lines 5-50) and graphic data of 
two or more graphic images to be overlaid on two or more video objects or entities in 
the corresponding video frame (Column 5, line 54-65, Column 6, lines 8-19, Column 7, 
lines 33-47); and 

an augmented video information transmission generator receiving the annotation 
data, the video information, and the annotation timing information, the augmented video 
information transmission generator generating an augmented video transmission signal 
comprising annotation data, the annotation data timing information, and the video 
information, transmits the augmented video transmission signal to a receiver (Column 5, 
lines 54-65, Column 6, lines 29-33, Column 11, lines 36-37, Figure 18, Column 3, liens 
53-55, Column 4, lines 3-7, 17-24), 

wherein the augmented video information transmission generator associates the 
video information with the annotation data using the annotation data timing information 
(Column 5, lines 54-65, Column 6, lines 29-33, Column 11, lines 36-37), 

wherein the receiver receiving the augmented video transmission signal is 
program for each of the plurality of consecutive video frames (Column 36, lines 41-49, 
Column 21 , lines 34-52) to: 

compare a current annotation data timing information with a current frame time of 
a current video frame (Abstract, Column 4, lines 35-37, 55-62, Column 5, line 54-65, 
Column 6, lines 8-19, Column 7, lines 33-47, Column 12, lines 21-46, Column 36, lines 
41-49, Column 21, lines 34-52); 
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retrieve the location and graphics data of the two or more graphics images from 
the identified mask if the comparison results in a match (Abstract, Column 4, lines 35- 
37, 55-62, Column 5, line 54-65, Column 6, lines 8-19, Column 7, lines 33-47, Column 
12, lines 21-46, Column 36, lines 41-49, Column 21, lines 34-52); and 

overlay the two or more graphics images on the corresponding two or more video 
object appearing in the video frame based on the retrieved location and graphics data, 
wherein the overlaying of the two or more graphics images is synchronized on a frame 
by frame basis with the movement of two or more video objects from the first location to 
the second location over the plurality of consecutive video frames (Abstract, Column 4, 
lines 35-37, 55-62, Column 5, line 54-65, Column 6, lines 8-19, Column 7, lines 21-47, 
Column 12, lines 21-46, Column 36, lines 41-49, Column 21, lines 34-52). 

Regarding Claims 2 and 36, Srinivasan discloses all the limitations of Claims 1 
and 35 respectively. Srinivasan discloses the augmented video information 
transmission generator comprises a component, which inserts data into the VBI or a 
vertical blanking insertion device (Column 35, lines 60-62, Column 36, lines 36-40). 

Regarding Claims 3 and 37, Srinivasan discloses all the limitations of Claims 1 
and 35 respectively. Srinivasan discloses that the augmented video information 
transmission generator comprises a digital video data multiplexer (Column 36, lines 25- 
32). 

Regarding Claims 4 and 38, Srinivasan discloses all the limitations of Claims 1 
and 35 respectively. Srinivasan discloses that he timing information comprises at least 
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one of timestamp information and a frame number information (Column 3, Column 4, 
lines 3-16, Column 7, line 23, Column 8, lines 5-14). 

Regarding Claims 5 and 39, Srinivasan discloses all the limitations of Claims 1 
and 35 respectively. Srinivasan discloses the programs comprise digital video signal 
(Column 36, Hnes 25-27). 

Regarding Claims 6 and 40, Srinivasan discloses all the limitations of Claims 1 
and 35 respectively. Srinivasan discloses the programs comprise analog video signal 
(Column 36, lines 36-40). 

Regarding Claims 7, Srinivasan discloses all the limitations of Claim 1 . 
Srinivasan disclose a post production environment (Figure .1,11, Figure 18, 251, 253, 
255), and a headend comprising the augmented video information transmission 
generator (Column 6, lines 33-35, Figure 18, 259), the video information and annotation 
data timing information are combined by the post production environment and 
transmitted to the headend (Column 7, lines 20-48, Column 8, lines 5-49, Column 35, 
lines 20-36). 

Regarding Claim 14, Srinivasan discloses all the limitations of Claims 1 and 35 
respectively. Srinivasan discloses displaying annotation data in response to a viewer 
request (Column 6, lines 8-19, Column 12, lines 21-40). 

Regarding Claims 17 and 46, Srinivasan discloses all the limitations of Claims 1 
and 35 respectively. Srinivasan discloses that the mask comprises location information 
of two or more objects or entities in an annotated video frame in the corresponding 
video frame (Column 5, lines 54-65, Column 7, lines 22-48, Column 8, lines 5-50). 
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Regarding Claims 18 and 47, Srinivasan discloses all the limitations of Claims 17 
and 46 respectively. Srinivasan discloses the location information includes a graphics 
location reference that represents a fixed relation to a set of pixels associated with each 
object (Column 9, lines 19-55, Figure 4). 

Regarding Claims 19 and 48, Srinivasan discloses all the limitations of Claims 18 
and 47 respectively. Srinivasan discloses a table of pixels based on a tracking box and 
object (Figure 4). Therefore, it is necessarily included that the pixels include the upper 
left most pixel in the associated pixel set if the object that is being tracked is located at 
the upper most left corner (Figure 4, Figure 3). 

Regarding Claims 20 and 49, Srinivasan discloses all the limitations of claims 18 
and 48 respectively. Srinivasan discloses a table of pixels based on a tracking box and 
object (Figure 4), if the object is in the center position then the centroid pixel is in the 
associated pixel set (Figure 4, Figure 3). Therefore, it is necessarily included that the 
pixels include the centroid pixel in the associated pixel set. 

Regarding Claim 41, Srinivasan discloses all the limitations of Claim 35. 
Srinivasan discloses inserting the annotation data timing information in a vertical 
blanking interval of an analog video signal (Column 36, lines 36-41). 

Regarding Claim 43, Srinivasan discloses all the limitations of Claim 35. 
Srinivasan discloses the transmitting the timing information and video information to a 
broadcast network and subsequently to the augmented video transmission generator 
(Column 5, lines 54-65, Column 6, lines 29-40, Column 7, lines 22-25, Column 36, lines 
25-40). 



Application/Control Number: 09/943,583 
Art Unit: 2623 



Page 1 1 



Claim Rejections - 35 USC § 103 

11. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 1 02 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

12. Claims 8, 9 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Srinivasan in view of Shoff et al (US 6,240,555 and hereafter referred to as "Shoff"). 

Regarding Claims 8, Srinivasan discloses all the limitations of Claim 7. 
Srinivasan does not disclose that the headend is a cable headend. In analogous art, 
Shoff discloses a post production environment (Figure 2, 22), a broadcast network 
(Column 4, lines 43-50), and the post production environment combines video data and 
synchronized timing data and transmits to a headend or node (Column 4, lines 43-50). 
Shoff discloses that the node is a cable headend (Column 4, lines 45-52). Therefore, it 
would have been obvious at the time the invention was made to one of ordinary skill in 
the art to modify Srinivasan to include that the headend is a cable headend (Column 4, 
lines 45-52) as taught by Shoff in order to enable viewer interactively with video 
program (Column 1 , lines 8-14) as disclosed by Shoff. 

Regarding Claims 9, Srinivasan discloses all the limitations of Claim 7. 
Srinivasan. does not to disclose the headend is a satellite headend. In analogous art, 
Shoff discloses a post production environment (Figure 2, 22), a broadcast network 
(Column 4, lines 43-50), and the post production environment combines video data and 
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synchronized timing data and transmits to a headend or node (Column 4, lines 43-50). 
Shoff discloses that the node is a satellite headend (Column 4, lines 45-52). Therefore, 
it would have been obvious at the time the invention was made to one of ordinary skill in 
the art to modify Srinivasan to include that the headend is a satellite headend (Column 
4, lines 45-52) as taught by Shoff in order to enable viewer interactively with video 
program (Column 1, lines 8-14) as disclosed by Shoff. 

13. Claims 21, 50-52, 55, 56 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Srinivasan in view of Wistendahl et al (US 6,496,981 and hereafter 
referred to as "Wistendahl") 

Regarding Claims 21 and 50, Shoff discloses all the limitations of Claims 1 and 
35 respectively. Srinivasan discloses that the mask comprises location information 
about an object in a video frame to be annotated or supplemental content to be added 
to video (Figure 4). Srinivasan is silent on the shape information. In analogous art, 
Wistendahl discloses that location and shape information of an object in the video frame 
(Column 10, lines 8-35). Therefore, it would have been obvious at the time the 
invention was made to one of ordinary skill in the art to modify Shoff to include shape 
information of an object in the video frame (Column 10, lines 8-35) as taught by 
Wistendahl in order to use media content for interactive television (Column 1, lines 12- 

14, 45-67) as disclosed by Wistendahl. 

Regarding Claim 51 , Srinivasan and Wistendahl disclose all the limitations of 
Claim 50. Wistendahl discloses the shape information is represented by a hyper link 
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(Column 10, lines 36-56). The hyperlink can be a graphical overlay of the object 
(Column 9, lines 28-33). 

Regarding Claim 52, Srinivasan and Wistendahl disclose all the limitations of 
Claim 50. Wistendahl discloses the shape information is represented by an outline of 
the object (Column 10, lines 25-28). 

Regarding Claim 55, Srinivasan discloses all the limitations of Claim 1 . 
Srinivasan is silent on the annotation data includes a plurality of object data packets, the 
mask including an identifier to an object mapping table including one or more of the 
plurality of object data packets, the object mapping table including at least one entry 
with an indicia from the corresponding mask identifying a particular video object, the 
entry further associating the indicia to information data structures included in one ore 
more of the plurality object data packets, the information data structures including 
information for the particular video object. Wistendahl discloses on annotation data 
further includes a plurality of object data packets (Figure 3, 32, Column 17, lines 8-10), 
the mask further including an identifier or a hyperlink to an object mapping table or N 
data information (including values for several objects) (Column 6, lines 17-38, Column 
7, lines 1-10) such as including at least one entry with an indicia from the corresponding 
mask identifying a particular video object (Figure 2, Column 6, lines 17-38), the entry 
further associating the indicia to information data structures included in one or more of 
the plurality of object data packets (Figure 2, Figure 3), the information data structures 
including information for particular video object (Column 9, lines 59-67, Column 10, lines 
1-56). Therefore, it would have been obvious at the time the invention was made to one 
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of ordinary skill in the art to modify Srinivasan to include annotation data further includes 
a plurality of object data packets (Figure 3, 32, Column 17, lines 8-10), the mask further 
including an identifier or a hyperlink to an object mapping table or N data information 
(including values for several objects) (Column 6, lines 17-38, Column 7, lines 1-10) such 
as including at least one entry with an indicia from the corresponding mask identifying a 
particular video object (Figure 2, Column 6, lines 17-38), the entry further associating 
the indicia to information data structures included in one or more of the plurality of 
object data packets (Figure 2, Figure 3), the information data structures including 
information for particular video object (Column 9, lines 59-67, Column 10, lines 1-56) as 
taught by Wistendahl in order to use media content for interactive television (Column 1 , 
lines 12-14, 45-67) as disclosed by Wistendahl. 

Regarding Claim 56, Srinivasan and Wistendahl disclose all the limitations of 
Claim 55. Wistendahl discloses the receiver being configured to overlay a graphics 
image on a particular video frame for the particular video object based on the graphics 
data included in the corresponding mask (Column 9, lines 59-67, Column 10, lines 1- 
56), retrieve the identifier of the object mapping table from the corresponding mask 
response to a user selection associated with the overlaid graphics image (Column 6, 
lines 60-67, Column 7, lines 1-10, Column 9, lines 59-67, Column 10, lines 1-56); 
retrieve the object mapping table based on the retrieved identifier (Figure 3, Figure 7a); 
identify the indicia in the corresponding mask for the particular video object for which 
the graphics image was overlaid (Column 15, lines 3-5); locate the entry in the object 
mapping table with the identified indicia (Figure 5b, 51b, Figure 7a, Figure 7B); identify 
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the information data structures associated with the located entry (Figure 5a, Figure 5b; 
retrieve the information in the identified information data structures; and display the 
retrieved information on the display device (Figure 7a). 

14. Claim 42 is rejected under 35 U.S.C. 103(a) as being unpatentable over 
Srinivasan in view of Oguro et al (US 2001/0033739 and hereafter referred to as 
"Oguro"). 

Regarding Claim 42, Srinivasan discloses all the limitations of Claim 35. 
Srinivasan disclose inserting timing information into the VBI of an analog signal (Column 
36, lines (Column 25-40). Srinivasan is silent the insertion of data in the vertical 
ancillary data or VBI of a digital video signal. Oguro discloses a television broadcast 
system, which transmits digital video signals to the user (Page 5, paragraph 0077) and 
inserts data into the VBI of a digital video signal (Page 5, paragraph 0076). Therefore, it 
would have been obvious at the time the invention was made to one of ordinary skill in 
the art to modify Srinivasan to include inserting timing information or time stamps in the 
VBI (Column 4, lines 36, 37, 40-55) as taught by Hidary in order to provide a user 
friendly visual experience of television programming to a viewer (Column 1, lines 53-62) 
as disclosed by Hidary. Therefore, it would have been obvious at the time the invention 
was made to one of ordinary skill in the art to modify Srinivasan to include inserting 
information in the VBI of a digital video signal (Page 5, paragraph 0076) as taught by 
Oguro in order to provide copy protect television broadcast programs of a digital video 
signal (Page 1, paragraphs 0001-0006) as disclosed by Oguro. 
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15. Claims 61, 62 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Srinivasan in view of Kaiser et al (US 6,615,408 and hereafter referred to as "Kaiser"). 

Regarding Claims 61 and 62, Srinivasan discloses all the limitations of Claims 1 
and 35 respectively. Srinivasan discloses overlaying two or more graphics images for 
alerting a viewer of the interactive data (Column 6, liens 6-19). Srinivasan is silent on 
the overlaying of the graphics images is for alerting a viewer of the interactive data 
associated prior to the viewer transmitting an interactive command with respect to one 
of video object or video objects. Kaiser discloses overlaying of the two or more 
graphics images is for alerting a viewer of the interactive data associated with the two or 
more video objects prior to the viewer transmitting an interactive command with respect 
to one of the two or more video objects (Figure 6B, Column 10, lines 20-41). Therefore, 
it would have been obvious at the time the invention was made to one of ordinary skill in 
the art to modify Srinivasan to include overlaying of the two or more graphics images is 
for alerting a viewer of the interactive data associated with the two or more video 
objects prior to the viewer transmitting an interactive command with respect to one of 
the two or more video objects (Figure 6B, Column 10, lines 20-41) as taught by Kaiser 
in order display a locator or locators for a video object or objects without clutter video 
(Column 1 , lines 8-11, 34-40) as disclosed by Kaiser and to make it easier for a user so 
that they do not have to determine what is interactive and provides more information. 
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Conclusion 

16. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Farzana E. Hossain whose telephone number is 571- 

272- 5943. The examiner can normally be reached on Monday to Friday 8:00 am to 
4:30 pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Christopher Grant can be reached on 571-272-7294. The fax phone 
number for the organization where this application or proceeding is assigned is 571- 

273- 8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 

CHRIS KELLEY 
SUPERVISORY PATENT EXAMINER 
TECHNOLOGY CENTER 2600 
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